1. Technical Field
The present invention relates to an image processing apparatus, an image processing method, a program and a semiconductor integrated circuit for reconstructing a high-resolution digital image signal by performing super-resolution interpolation on image signals.
2. Background Art
Recently, methods for converting a sequence of low-resolution images into a high-resolution image or a sequence of high-resolution images have attracted considerable interest among computer scientists and image processing specialists. These methods are commonly referred to as super-resolution, super-resolution interpolation or super-resolution reconstruction.
The basic idea behind super-resolution is to exploit motions in low-resolution images at sub-pixel level in order to reconstruct image details that are not apparent from any one of these images by itself.
Super-resolution techniques are particularly interesting in the context of image acquisition, since they provide an efficient method to improve image resolution without employing costly high-performance imaging devices.
FIG. 1A is a block diagram of a conventional image acquisition system.
As shown in FIG. 1A, a sampling unit 120 samples an input image 101 at a predetermined sampling frequency. The processing/recording unit 150 processes it or records it onto a recording medium. In the case where the input image contains video frequencies which are higher than the Nyquist frequency of the sampling unit 120, aliasing, that is, folding noise occurs in the sampled image. This can be avoided by an anti-aliasing filter (folding noise prevention filter) 110 as shown in FIG. 1B. The anti-aliasing filter 110 is a low-pass filter which removes video frequencies exceeding the Nyquist frequency before the sampling. Hence, the conventional image acquisition and reproduction apparatus shown in FIG. 1B outputs an image 190 free of folding noise.
Functions of a conventional anti-aliasing filter 110 are described with reference to FIGS. 2A to 2C and FIGS. 3A to 3E. For simplification, a signal is assumed to be a one-dimensional signal here. In each of the drawings, the left-hand graph shows a signal in a spatial domain. The horizontal axis x shows one-dimensional spatial coordinate or a time axis, and the longitudinal axis shows luminance. Likewise, the right-hand graph shows a signal transformed (Fourier-transformed) into a frequency domain. The horizontal axis ω shows frequency (radian), the longitudinal axis shows frequency strength, and ωN shows the Nyquist frequency.
FIG. 2A shows a rapidly varying video signal with an accordingly broad frequency spectrum F. Sampling this signal can be expressed as a multiplication with a Dirac comb g as represented schematically in FIG. 2B. Note that the Fourier transform of a Dirac comb is also a Dirac comb G. Since multiplication of two signals in the spatial domain corresponds to a convolution of the transformed signals in the frequency domain, the spectrum of the sampled signal F*G takes the form as indicated on the right-hand side of FIG. 2C. In other words, the spectrum F*G of the sampled signal (shown as solid lines) is the sum of the spectra (shown as dashed lines) transformed and replicated periodically.
As shown in FIG. 2C, the transformed and replicated spectra (shown as dashed lines) overlap with each other. The spectral power of the sampled signal at a certain frequency is thus contaminated by contributions from other frequencies that are a so-called alias to the certain frequency. In the spatial domain, aliasing artifacts become clear and apparent noise such as Moiré patterns or jaggy which occurs along smooth edge line portions.
In order to prevent aliasing, it is thus necessary to prevent overlapping of the spectra in the sampling. This can be achieved by band-limiting the initial signal by means of a low-pass filter hAA as shown in FIG. 3B. The anti-aliasing filter 110 has characteristics of the low-pass filter hAA.
FIG. 3A represents the video signal f, and FIG. 3B represents the low-pass filter hAA used for band-limiting the signal by convolving the signal in the spatial domain or multiplying the signal in the frequency domain. FIG. 3C shows the result f*hAA of the low-pass filtering (shown as a solid line) in comparison to the video signal f (shown as a dashed line).
As explained above, sampling corresponds to multiplication of the signal with a Dirac comb g in the spatial domain, and to a convolution with the corresponding Dirac comb G in the frequency domain (cf. FIG. 3D). Since the spectrum of the signal has been band-limited, the transformed and replicated spectra F*HAA do no longer overlap with each other (cf. FIG. 3E), so that no aliasing occurs.
Next, conventional image acquisition systems with super-resolution interpolation are illustrated. FIG. 4A is a block diagram of the configuration of the conventional image acquisition system including the sampling unit 120, the processing/recording unit 150, and the super-resolution interpolation unit 160. The input image 101 is sent to the sampling unit 120. The sampling unit 120 generates a digital image by sampling the input image 101 at a predetermined sampling frequency. The processing/recording unit 150 outputs, as a low-resolution output image 191, the digital image at a sampling resolution of the original image. Otherwise, the low-resolution output image is outputted to the super-resolution interpolation unit 160. The super-resolution interpolation unit 160 outputs a high-resolution output image 192 having a resolution which is higher than the original by performing super-resolution interpolation on the low-resolution output image.
FIG. 4B is a block diagram of a conventional image acquisition system similar to the system shown in FIG. 4A but with an additional anti-aliasing filter 110. The system of FIG. 4B is similar to that of FIG. 4A except that the input images are filtered by an anti-aliasing filter 110 before sampling. Hence, the image quality of the low-resolution output images 191 is enhanced since folding noise is removed. However, due to the anti-aliasing filtering, the high-resolution images 192 which are outputted by the super-resolution interpolation unit 160 do not contain any finer image details than in the low-resolution images.
Super resolution is described with reference to FIG. 5. A conventional super-resolution reconstruction method basically includes two steps. In a first step, motion estimation and registration are performed on the input images. Motion estimation is to estimate a motion in each reference image with respect to corresponding current low-resolution image with sub-pixel precision. Registration is to register reference images on a high-resolution grid 520 corresponding to the current low-resolution image using the estimated motion.
In the second step, nonuniform interpolation techniques can be employed to obtain interpolated values for each point of the high-resolution grid 520 so as to produce a reconstructed high-resolution image 530. This is disclosed in, for example, Patent Reference 1.    Patent Reference 1: Japanese Laid-open Patent Application Publication No. 2000-339450.